{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Ejercicio Feature Engineering\n", "\n", "### Reemplace todos los valores nulos o vacíos del archivo csv pertinente\n", "\n", "Este ejercicio consiste en cargar un archivo csv que tiene valores nulos en más de una columna. Es requerido reemplazar los valores nulos por el mejor valor posible." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. [Importar las librerias y cargar el archivo csv](#1)\n", "2. [Visualizar que datos son los que faltan](#2)\n", "3. [Ver los filas de los datos faltantes y la matriz de correlación](#3)\n", "4. [Reemplazar los valores nulos de la columna mpg](#4)\n", "5. [(Opcional) Reemplazar los valores nulos de mpg utilizando funciones programadas por mí](#5)\n", "6. [Reemplazar los valores nulos de la columna caballos_potencia](#6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Importar las librerias y cargar el archivo csv\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import math\n", "\n", "autos = pd.read_csv(os.path.join('./csv/', 'datos_automoviles.csv'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Visualizar que datos son los que faltan\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nombrecilindrospesoanioterritorioaceleracionmpgcaballos_potenciadesplazamiento
count406406.000000406.000000406.000000406406.000000398.000000400.000000406.000000
unique312NaNNaNNaN3NaNNaNNaNNaN
topFord PintoNaNNaNNaNUSANaNNaNNaNNaN
freq6NaNNaNNaN254NaNNaNNaNNaN
meanNaN5.4753692979.4137931975.921182NaN15.51970423.514573105.082500194.779557
stdNaN1.712160847.0043283.748737NaN2.8033597.81598438.768779104.922458
minNaN3.0000001613.0000001970.000000NaN8.0000009.00000046.00000068.000000
25%NaN4.0000002226.5000001973.000000NaN13.70000017.50000075.750000105.000000
50%NaN4.0000002822.5000001976.000000NaN15.50000023.00000095.000000151.000000
75%NaN8.0000003618.2500001979.000000NaN17.17500029.000000130.000000302.000000
maxNaN8.0000005140.0000001982.000000NaN24.80000046.600000230.000000455.000000
\n", "
" ], "text/plain": [ " nombre cilindros peso anio territorio \\\n", "count 406 406.000000 406.000000 406.000000 406 \n", "unique 312 NaN NaN NaN 3 \n", "top Ford Pinto NaN NaN NaN USA \n", "freq 6 NaN NaN NaN 254 \n", "mean NaN 5.475369 2979.413793 1975.921182 NaN \n", "std NaN 1.712160 847.004328 3.748737 NaN \n", "min NaN 3.000000 1613.000000 1970.000000 NaN \n", "25% NaN 4.000000 2226.500000 1973.000000 NaN \n", "50% NaN 4.000000 2822.500000 1976.000000 NaN \n", "75% NaN 8.000000 3618.250000 1979.000000 NaN \n", "max NaN 8.000000 5140.000000 1982.000000 NaN \n", "\n", " aceleracion mpg caballos_potencia desplazamiento \n", "count 406.000000 398.000000 400.000000 406.000000 \n", "unique NaN NaN NaN NaN \n", "top NaN NaN NaN NaN \n", "freq NaN NaN NaN NaN \n", "mean 15.519704 23.514573 105.082500 194.779557 \n", "std 2.803359 7.815984 38.768779 104.922458 \n", "min 8.000000 9.000000 46.000000 68.000000 \n", "25% 13.700000 17.500000 75.750000 105.000000 \n", "50% 15.500000 23.000000 95.000000 151.000000 \n", "75% 17.175000 29.000000 130.000000 302.000000 \n", "max 24.800000 46.600000 230.000000 455.000000 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "autos.describe(include='all')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Ver los filas de los datos faltantes y la matriz de correlación\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nombrecilindrospesoanioterritorioaceleracionmpgcaballos_potenciadesplazamiento
10Citroen Ds-21 Pallas430901970Europe17.5NaN115.0133.0
11Chevrolet Chevelle Concours (Sw)841421970USA11.5NaN165.0350.0
12Ford Torino (Sw)840341970USA11.0NaN153.0351.0
13Plymouth Satellite (Sw)841661970USA10.5NaN175.0383.0
14Amc Rebel Sst (Sw)838501970USA11.0NaN175.0360.0
17Ford Mustang Boss 302833531970USA8.0NaN140.0302.0
39Volkswagen Super Beetle 117419781971Europe20.0NaN48.097.0
367Saab 900S428001981Europe15.4NaN110.0121.0
\n", "
" ], "text/plain": [ " nombre cilindros peso anio territorio \\\n", "10 Citroen Ds-21 Pallas 4 3090 1970 Europe \n", "11 Chevrolet Chevelle Concours (Sw) 8 4142 1970 USA \n", "12 Ford Torino (Sw) 8 4034 1970 USA \n", "13 Plymouth Satellite (Sw) 8 4166 1970 USA \n", "14 Amc Rebel Sst (Sw) 8 3850 1970 USA \n", "17 Ford Mustang Boss 302 8 3353 1970 USA \n", "39 Volkswagen Super Beetle 117 4 1978 1971 Europe \n", "367 Saab 900S 4 2800 1981 Europe \n", "\n", " aceleracion mpg caballos_potencia desplazamiento \n", "10 17.5 NaN 115.0 133.0 \n", "11 11.5 NaN 165.0 350.0 \n", "12 11.0 NaN 153.0 351.0 \n", "13 10.5 NaN 175.0 383.0 \n", "14 11.0 NaN 175.0 360.0 \n", "17 8.0 NaN 140.0 302.0 \n", "39 20.0 NaN 48.0 97.0 \n", "367 15.4 NaN 110.0 121.0 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "autos.loc[autos['mpg'].isnull()]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nombrecilindrospesoanioterritorioaceleracionmpgcaballos_potenciadesplazamiento
38Ford Pinto420461971USA19.025.0NaN98.0
133Ford Maverick628751974USA17.021.0NaN200.0
337Renault Lecar Deluxe418351980Europe17.340.9NaN85.0
343Ford Mustang Cobra429051980USA14.323.6NaN140.0
361Renault 18I423201981Europe15.834.5NaN100.0
382Amc Concord Dl430351982USA20.523.0NaN151.0
\n", "
" ], "text/plain": [ " nombre cilindros peso anio territorio aceleracion \\\n", "38 Ford Pinto 4 2046 1971 USA 19.0 \n", "133 Ford Maverick 6 2875 1974 USA 17.0 \n", "337 Renault Lecar Deluxe 4 1835 1980 Europe 17.3 \n", "343 Ford Mustang Cobra 4 2905 1980 USA 14.3 \n", "361 Renault 18I 4 2320 1981 Europe 15.8 \n", "382 Amc Concord Dl 4 3035 1982 USA 20.5 \n", "\n", " mpg caballos_potencia desplazamiento \n", "38 25.0 NaN 98.0 \n", "133 21.0 NaN 200.0 \n", "337 40.9 NaN 85.0 \n", "343 23.6 NaN 140.0 \n", "361 34.5 NaN 100.0 \n", "382 23.0 NaN 151.0 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "autos.loc[autos['caballos_potencia'].isnull()]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Se quiere saber que variable(columna) tiene la mejor correlación con la variable que tiene valores nulos\n", "en este caso, la variable que tiene la mejor correlación con mpg es peso.\n", "\n", "Pero, como peso puede adqurir múltiples valores es mejor utilizar a la variable cilindros,\n", "porque cilindros sólo puede adqurir pocos valores, por tanto, el valor de la mediana será más preciso." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cilindrospesoanioaceleracionmpgcaballos_potenciadesplazamiento
cilindros1.0000000.895220-0.360762-0.522452-0.7753960.8441580.951787
peso0.8952201.000000-0.315389-0.430086-0.8317410.8665860.932475
anio-0.360762-0.3153891.0000000.3019920.579267-0.424419-0.381714
aceleracion-0.522452-0.4300860.3019921.0000000.420289-0.697124-0.557984
mpg-0.775396-0.8317410.5792670.4202891.000000-0.778427-0.804203
caballos_potencia0.8441580.866586-0.424419-0.697124-0.7784271.0000000.898326
desplazamiento0.9517870.932475-0.381714-0.557984-0.8042030.8983261.000000
\n", "
" ], "text/plain": [ " cilindros peso anio aceleracion mpg \\\n", "cilindros 1.000000 0.895220 -0.360762 -0.522452 -0.775396 \n", "peso 0.895220 1.000000 -0.315389 -0.430086 -0.831741 \n", "anio -0.360762 -0.315389 1.000000 0.301992 0.579267 \n", "aceleracion -0.522452 -0.430086 0.301992 1.000000 0.420289 \n", "mpg -0.775396 -0.831741 0.579267 0.420289 1.000000 \n", "caballos_potencia 0.844158 0.866586 -0.424419 -0.697124 -0.778427 \n", "desplazamiento 0.951787 0.932475 -0.381714 -0.557984 -0.804203 \n", "\n", " caballos_potencia desplazamiento \n", "cilindros 0.844158 0.951787 \n", "peso 0.866586 0.932475 \n", "anio -0.424419 -0.381714 \n", "aceleracion -0.697124 -0.557984 \n", "mpg -0.778427 -0.804203 \n", "caballos_potencia 1.000000 0.898326 \n", "desplazamiento 0.898326 1.000000 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "autos.corr()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Reemplazar los valores nulos de la columna mpg\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "scrolled": false }, "outputs": [], "source": [ "cilindros_agrupacion = autos.groupby('cilindros')\n", "\n", "# Esto crea una serie del mismo tamaño del DataFrame que contiene el valor de la mediana de las mpg con respecto\n", "# a los cilindros.\n", "mediana_mpg_por_cilindro = cilindros_agrupacion['mpg'].transform('median')\n", "\n", "# Reemplazar los valores nulos de mpg con la mediana obteniad anteriormente\n", "autos['mpg'].fillna(mediana_mpg_por_cilindro, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. (Opcional) Reemplazar los valores nulos de mpg utilizando funciones programadas por mí\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "cilindros_agrupacion = autos.groupby('cilindros')\n", "# Crear una serie que tenga como índices los cilindros y como valor la mediana de las mpg seǵun los cilindros\n", "mediana_mpg_segun_cilindros = cilindros_agrupacion['mpg'].median()\n", "\n", "# Funcion que recibe una fila y si tiene la columna mpg como nan le pone el valor de la mediana de las mpg\n", "# según el cilindro\n", "def poner_mediana_en_mpg_nulos(fila):\n", " if math.isnan(fila['mpg']):\n", " fila['mpg'] = mediana_mpg_segun_cilindros[fila['cilindros']]\n", " return fila\n", "\n", "# Aplicar a cada fila del data frame la funcion definida arriba\n", "autos_con_mpg = autos.apply(poner_mediana_en_mpg_nulos, axis=1)\n", "\n", "# La misma funcion pero utilizando cálculo lambda\n", "#autos_con_mpg = autos.apply(lambda x: mediana_mpg_segun_cilindros[x['cilindros']] if math.isnan(x['mpg']) else x, axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Reemplazar los valores nulos de la columna caballos_potencia\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# El procedimiento es el mismo que el de reemplazar los valores nulos de mpg\n", "mediana_caballos_potencia_segun_cilindros = cilindros_agrupacion['caballos_potencia'].transform('median')\n", "autos['caballos_potencia'].fillna(mediana_caballos_potencia_segun_cilindros, inplace=True)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 2 }